A Real-time augmented reality robot integrated with artificial intelligence for skin tumor surgery - experimental study and case series.

BACKGROUND: Skin tumors affect many people worldwide, and surgery is the first treatment choice. Achieving precise preoperative planning and navigation of intraoperative sampling remains a problem and is excessively reliant on the experience of surgeons, especially for Mohs surgery for malignant tumors. MATERIALS AND METHODS: To achieve precise preoperative planning and navigation of intraoperative sampling, we developed a real-time augmented reality (AR) surgical system integrated with artificial intelligence (AI) to enhance three functions: AI-assisted tumor boundary segmentation, surgical margin design, and navigation in intraoperative tissue sampling. Non-randomized controlled trials were conducted on manikin, tumor-simulated rabbits, and human volunteers in xxx Laboratory to evaluate the surgical system. RESULTS: The results showed that the accuracy of the benign and malignant tumor segmentation were 0.9556 and 0.9548, respectively, and the average AR navigation mapping error was 0.644 mm. The proposed surgical system was applied in 106 skin tumor surgeries, including intraoperative navigation of sampling in 16 Mohs surgery cases. Surgeons who have used this system highly recognize it. CONCLUSIONS: The surgical system highlighted the potential to achieve accurate treatment of skin tumors and to fill the gap in global research on skin tumor surgery systems.

Feature pyramid self-attention network for respiratory motion prediction in ultrasound image guided surgery.

PURPOSE: The robot-assisted automated puncture system under ultrasound guidance can well improve the puncture accuracy in ablation surgery. The automated puncture system requires advanced definition of the puncture location, while the displacement of thoracic-abdominal tumors caused by respiratory motion makes it difficult for the system to locate the best puncture position. Predicting tumor motion is an effective way to help the automated puncture system output a more accurate puncture position. METHODS: In this paper, we propose a self-attention-based feature pyramid algorithm FPSANet for time-series forecasting, which can extract both linear and nonlinear dependencies of time series. Firstly, we use the temporal convolutional network as the backbone to extract different scale time-series features, and the self-attention module is followed to weigh more significant features to improve nonlinear prediction. Secondly, we use autoregressive models to perform linear prediction. Finally, we directly combine the above two kinds of predictions as the final prediction. RESULTS: FPSANet is trained and tested on our private datasets captured from clinical individuals, and we predict the target position after 50 ms, 150 ms, 300 ms and 400 ms. The result shows the evaluation criteria of the MAE is less than 1 mm at 50 ms and 150 ms, and less than 2 mm at 300 ms. Compared with the AR model, bidirectional LSTM and RVM, our method not only outperforms both models in accuracy (AR: ~ 7.7%; bidirectional LSTM: ~ 75.9%; RVM: ~ 76.5%) but is also more stable on different types of respiratory curves. CONCLUSION: Respiratory motion in the liver in actual clinical practice vary widely from person to person, while sometimes having less distinct periodic patterns. Under these conditions, our algorithm has the advantage of excellent stability for prediction on various sequences, and its running time of performing single sequence prediction can meet clinical requirements.

FPSN-FNCC: an accurate and fast motion tracking algorithm in 3D ultrasound for image-guided interventions.

The uncertain motions of a target caused by the breath, heartbeat and body drift of a patient can increase the target locating error during image-guided interventions, and that may cause additional surgery trauma. A surgery navigation system with accurate motion tracking is important for improving the operation accuracy and reducing trauma. In this work, we propose an accurate and fast tracking algorithm in three-dimensional (3D) ultrasound (US) sequences for US-guided surgery to achieve moving object tracking. The idea of this algorithm is as follows. Firstly, feature pyramid architecture is introduced into a Siamese network to extract multiscale convolutional features. Secondly, to improve the network discriminative power and the robustness to ultrasonic noise and gain variation, we use the normalized cross correlation (NCC) to calculate the similarity between template block and search block. Thirdly, a fast NCC (FNCC) is proposed, which can perform the real-time tracking. Finally, a density peaks clustering approach is used to compensate the motion of the target and further improve the tracking accuracy. The proposed algorithm is evaluated on a CLUST dataset that includes 22 sets of 3D US sequences, and the mean error of 1.60±0.97 mm compared with manual annotations is obtained. After comparing with other published works, the results show that our algorithm achieves the comparable performance. The ablation study proves that the results benefit from the feature pyramid architecture and FNCC. These findings show that our algorithm may improve the motion tracking accuracy in image-guided interventions.

Tracking 3D ultrasound anatomical landmarks via three orthogonal plane-based scale discriminative correlation filter network.

PURPOSE: In abdominal interventional therapy, accurate motion tracking of the target is regarded as crucial to minimize trauma and optimize dosage delivery. Meanwhile, three-dimensional (3D) ultrasound (US) is an attractive modality to show the real-time motion pattern of the target. In this work, we developed an accurate and robust landmark tracking algorithm for 3D US sequences. METHODS: The proposed algorithm introduces a three orthogonal planes (TOPs) based scale discriminative correlation filter network for 3D US landmarks tracking. First, we couple the fully convolutional network (FCN) with scale discriminative correlation filter (SDCF) to generate an effective tracker. And SDCF is reformulated as a differentiable layer, which ensures the network can perform scale learning and end-to-end training. Next, we train the end-to-end network on millions of natural images. Finally, we convert 3D US image to 2D three-channel image by TOP transformation and feed them to the proposed network for performing online tracking. RESULTS: Online tracking performance was evaluated on the Challenge of Liver Ultrasound Tracking (CLUST) dataset with 22 sets of 3D US sequences, obtaining mean error of 1.63 ± 1.04 mm and 95th percentile (95%ile) error of 3.37 mm, when compared with manual annotations annotated by surgeons. Ablation study indicates that the promising results benefit from SDCF and scale learning, which alleviates the influence from deformation. The findings of the clinical analysis support that the proposed algorithm can work well with different initial patch sizes, which means that our algorithm has potential to lighten the burden of surgeons. CONCLUSIONS: We propose a flexible, accurate and robust landmark tracking algorithm for the image-guided interventions, and our algorithm is comparable with the state-of-the-art approaches. The tracking accuracy and robustness show that our algorithm has potential in 3D US-guided abdominal interventional therapies. Furthermore, more researches are needed to improve the computing speed of the algorithm to achieve real-time tracking.